Gaussian Processes for Big Data through Stochastic Variational Inference

Gaussian processes [GP 10] are perhaps the dominant approach for inference on functions. They underpin a range of algorithms for regression, classification and unsupervised learning. Unfortunately, exact inference in a GP has complexity O(n) with storage demands of O(n) and this hinders application of these models for ‘big data’. Various approximate techniques have been suggested [see e.g. 1, 11, 9, 12] which lead to a computational complexity of O(nm) and storage demands of O(nm) where m is a user selected parameter governing a number of “inducing variables”. However, even the reduced storage requirements which are linear in the data set size are prohibitive for big data, where n can be many millions. For parametric models, stochastic gradient descent is often applied to resolve this storage issue, but in the GP domain, it hasn’t been clear how this should be performed. In this paper we show how recent advances in variational inference [5, 6] can be combined with the idea of inducing variables to develop a practical algorithm for fitting GPs based around stochastic variational inference (SVI).